Linode‘s recent network outage happened when I was in the middle of upgrading the OS. And when the issue got resolved, I found my VPS broken.
The full story
Soon after Ubuntu‘s new LTS version released, Linode Library published a guide explaining how to upgrade to Ubuntu 10.04 LTS. After one month, I decided to do it. And it was the same day, the network outage happened.
I was in the middle of upgrading. When I saw that my SSH terminal stopped responding, I thought that I made some mistake. I tried to login to Linode control panel, but the site was not resolving! I feared the worst.
I checked their twitter account, but couldn’t find anything about any outage. But I found a lot of tweeple complaining about it. After some time, there were an official tweet.
When the issue got resolved, I found that I can’t login via SSH. So, I tried the Ajax console from the Linode control panel. I got the following error:
init: ureadahead main process (977) terminated with status 5 mount: mount point /dev/pts does not exist mountall: mount /dev/pts [987] terminated with status 32 mountall: Filesystem could not be mounted: /dev/pts mount: mount point /dev/shm does not exist mountall: mount /dev/shm [988] terminated with status 32 mountall: Filesystem could not be mounted: /dev/shm mount: mount point /dev/pts does not exist mountall: mount /dev/pts [992] terminated with status 32 mountall: Filesystem could not be mounted: /dev/pts mount: mount point /dev/shm does not exist mountall: mount /dev/shm [993] terminated with status 32 mountall: Filesystem could not be mounted: /dev/shm init: mountall main process (982) terminated with status 4 Mount of root filesystem failed.
I opened a support ticket. Got a reply within 6 minutes.
The error that you are seeing is common with systems that have successfully upgraded, but have not had the needed changes made to their ‘/etc/fstab’. You can mount your disk image using the Finnix recovery image to make the needed configuration changes. More information on using Finnix can be found at: http://library.linode.com/troubleshooting/finnix-recovery
I created a finnix recovery profile and booted into it, but /mnt/xvdb/etc/fstab already had dev /dev devtmpfs rw 0 0 in it. But I saw a user comment on Linode library post saying that s/he had to use tmpfs instead of devtmpfs. I tried it and booted into normal profile. It worked.
Finally, I got my ubuntu upgraded to 10.04 LTS. But there were two issues. Apache showing a warning on start-up: [warn] NameVirtualHost *:80 has no VirtualHosts and MySQL was not running.
By googling, I found that this problem occurs when there are two instances of NameVirtualHost *:80 in the apache configuration. The new ports.conf already has this entry, so I commented out the entry I had in apache2.conf. That fixed the warning issue.
MySQL problem turned out to be another configuration change. MySQL 5.1 cannot have skip-bdb in my.cnf. Fixed it, and MySQL started running.
It was 2.52 AM when I went to sleep.
But all’s well that ends well. ;-) Learned a lot in the process. Happy about the prompt support by Linode. Not happy about the outages though.
